Holographic Reduced Representations I Introduction Ii.a Associative Memories Be the Trace Composition Operation. Let ~ Ii.c Convolution-correlation Memories

Associative memories are conventionally used to represent data with very simple structure: sets of pairs of vectors. This paper describes a method for representing more complex com-positional structure in distributed representations. The method uses circular convolution to associate items, which are represented by vectors. Arbitrary variable bindings, short sequences of various lengths, simple frame-like structures, and reduced representations can be represented in a xed width vector. These representations are items in their own right, and can be used in constructing com-positional structures. The noisy reconstructions extracted from convolution memories can be cleaned up by using a separate associative memory that has good reconstructive properties. Distributed representations 13] are attractive for a number of reasons. They ooer the possibility of representing concepts in a continuous space, they degrade gracefully with noise, and they can be processed in a parallel network of simple processing elements. However, the problem of representing compositional structure 1 in distributed representations has been for some time a prominent concern of both proponents and critics of connec-tionism 9, 32, 12]. Most work on neural-network style associative memories has focussed on either auto-associative or hetero-associative memories. Auto-associative memories, e.g., Hoppeld networks 14], store an unordered set of items. They can be used to recall item given a distorted version of one. Hetero-associative memories, e.g., holo-graphic memories and matrix memories 37, 8, 22, 5, 38], store a set of pairs of items. One item of a pair can be recalled using the other as a cue. Matrix style memories are the more popular class, owing to superior storage capacity and fewer constraints on vectors to be stored. For artiicial intelligence tasks such as language processing and reasoning the need arises to represent more 1 I.e., recursive, or tree-like structure. complex data structures such as sequences and trees. It is diicult to represent sequences or trees in distributed representations using associations of pairs (or even n-tuples) of items and retain the beneets of distributed representations. The problem with representing compositional structure in most associative memories is that items and associations are represented in diierent spaces. For example, in a Hoppeld memory (a matrix style memory) items are represented on unit activations (a vector) and associations are represented on connections weights (a matrix). This makes it diicult to represent relationships with recursive structure in which an association of items may be the subject of another association. Hinton 12] discusses this problem and …